Experience with a Distributed File System Implementation

This paper highlights some of the lessons learned during the course of implementing xFS, a fully distributed le system. xFS is an interesting case study for two reasons. First, xFS's serverless architecture leads to more complex distributed programming issues than are faced by traditional client-server operating system services. Second, xFS implements a complex, multithreaded service that is tightly coupled with the underlying operating system. This combination turned out to be quite challenging. On one hand, the complexity of the system forced us to turn to distributed programming tools based on formal methods to verify the correctness of our distributed algorithms; on the other hand the complex interactions with the operating system on individual nodes violated some of the tools' assumptions, making it di cult to use them in this environment. Furthermore, the xFS system tested the limits of abstractions such as threads, RPC, and vnodes that have traditionally been used in building distributed le systems. Based on our experience, we suggest several strategies that should be followed by those wishing to build distributed operating systems services, and we also indicate several areas where programming tools and operating system abstractions might be improved.